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11.3.4 Solved Problems 

Problem 1 

Consider a continuous-time Markov chain X{t) with the jump chain shown in Figure 11.25. 
Assume Ai = 2, A 2 = 3, and A 3 = 4. 



1 

2 

Figure 11.25 - The jump chain for the Markov chain of Problem 1 

a. Find the stationary distribution of the jump chain 7 7 = 7TT, 772 5 7^3 • 

b. Using 77 , find the stationary distribution for X(t). 

Note: Although this jump chain has self-transitions, you can still use the discussed methods. In 
fact, X{t) can be equivalently shown by a jump chain with no self-transitions along with 
appropriate holding times. 

• Solution 

o a. To find the stationary distribution of the jump chain, 77 = 77 1 ,772 , 7^3 > 
we need to solve 

1- 

TTl = 2 71 ” 3 ’ 

1 _ 1 _ 

7T2 = — 7Tl + -7T2 

1„ 2 _ 1_ 

7T3 = + -^772 + -773, 

7TT + 772 + 7T3 = 1- 
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We find 


TTl r 5 7T 2 ir’ ^"3 

15 15 


b. We have obtained 


S = —[4, 3, 8], 

We can find the limiting distribution of X(t ) using 


A, 


*3 


E 




keS \ k 


We obtain 


7Ti 


7T]_ 

Ai 


TTl | 7T2 | 71~3 
Ai A 2 A 3 


4 

2 


4 + 3 + 8 
2 3 4 


2 

5 


7T2 


7T2 

A 2 


TTl | 71~2 | 7<~3 
Ai A 2 A 3 


4 + 3 + 8 
2 3 4 


1 

5 
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7T3 


£3 

A 3 


7T]_ 7T2_ £3 

Ai A2 A3 


8 . 

4 


1 + 3 + 8 

2 3 4 


2 

5 


Thus, we conclude that 7T = -g- [2,1, 2] is the limiting distribution of 

-V( 0 . 


Problem 2 

Consider a continuous-time Markov chain X{t) that has the jump chain shown in Figure 
11.26 (this is the same Markov chain given in Example 11.19). Assume Ai = 2, A 2 = 1, 
and A 3 = 3. 

a. Find the generator matrix for this chain. 

b. Find the limiting distribution for X(t) by solving 7tG — 0. 

1 


1 


Figure 11.26 - The jump chain for the Markov chain of Problem 2 
• Solution 

o The jump chain is irreducible and the transition matrix of the jump chain is given 
by 
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0 10 


p = 


0 0 1 



The generator matrix can be obtained using 


We obtain 


9ij 


A iPij 

if 

—A* 

if i = j 

-2 2 

0 1 


G = 


0-11 



Solving 

7 tG — 0, and 7 ir + tt 2 + 773 = 1 

we obtain 7T — [3, 12,4], which is the same answer that we obtained in 

Example 11.19. 


Problem 3 

(A queuing system) Suppose that customers arrive according to a Poisson process with rate A 
at a service center that has a single server. Customers are served one at a time in order of 
arrival. Service times are assumed to be i.i.d. Exponential (/i) random variables and 
independent of the arrival process. Customers leave the system after being served. Our goal in 
this problem is to model the above system as a continuous-time Markov chain. Let X(t ) be 
the number of customers in the system at time t, so the state space is S — {0, 1,2, •••}. 
Assume 1 > 0. If the system is in state 1 at time t, then the next state would either be i + 1 
(if a new customers arrive) or state i — 1 (if a customer leaves). 

a. Suppose that the system is in state 0, so there are no customers in the system and the 
next transition will be to state 1. Let Tq be the time until the next transition. Show that 

T 0 rv -'' Exponential A). 
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b. Suppose that the system is currently in state 2, where 2 > 0. Let Tj be the time until 
the next transition. Show that ~ Exponential^ A + pi). 

c. Suppose that the system is at state i. Find the probability that the next transition will be 
to state 2 + 1. 

d. Draw the jump chain, and provide the holding time parameters A,;. 

e. Find the Generator matrix. 

f. Draw the transition rate diagram. 

• Solution 

o Note that to solve this problem, we use several results from the Poisson process 
section. In particular, you might want to review merging and splitting of Poisson 
processes before reading the solution to this problem. 

a. If there are no customers in the system, the next transition occurs when a 
new customer arrives. Since the customers arrive according to a Poisson 
process, and the interarrival times in the Poisson process have 
Exponential[X) distribution, we conclude 

T 0 r ^ j Exponential A). 

b. Suppose that the system is in state i, where i > 0. Thus, there is a 
customer being served. We assume the service times have 
Exponential (/x) distribution. The next transition occurs either when a 
new customer arrives, or when the service time of the current customer is 
ended. Thus, we can express T{ as 


Ti = min(I,y), 


C. 


where X ~ Exponential (A) andy ~ Exponential(p), and 
X and Y are independent. We claim that 

Ti ~ Exponential^ A + /X ). One way to see this is as follows. Here, 
you can imagine two independent Poisson processes. The first one is the 
customer arrival process. The second one is the process that has interarrival 
times equal to the service times. Now, the merged process has rate A + fi. 
Since T{ can be thought of the first arrival in the merged process, we 
conclude that Ti ~ Exponential A + /x). 

Suppose that the system is ate state i. We would like to find the probability 
that the next transition will be to state 2 + 1, shown by Pi,i+ 1 . Again 
consider the two Poisson processes defined above. We can model this 
system as follows. Each arrival in the merged process is of type 1 (customer 


arrival) with probability -r— , or of type 2 (customer departure) with 




A+/z 


probability . The probability p t ,i + \ is the probability that the first 

arrival in the merged process is of type 1. This happens with probability 

+^—, so we conclude 
A+/z 
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Pi,i +1 
Phi -1 


A 

A + fi 
1 — Pi,i +1 


P 

A + fi 


d. From the above, we can draw the jump chain as in Figure 11.27 


A A 

1 A A 



Figure 11.27 - The jump chain for the above queuing system. 


The holding time parameters, A,; 's, are given by 


Ao — A, 

Xi = A + /x, for i = 1,2, • • •. 


e. The generator matrix can be obtained using 


We obtain 


G = 


9ij — < 

XiPij 

if M J 



if * = j 

-A 

A 

0 

0 

P -(/i + A) 

A 

0 


0 ytt —(/i -f- A) A • • • 


f. Remember that in the transition rate diagram, the values Qij are shown on 
the edges (the values of ga's are not usually shown). The transition rate 
diagram for this chain is shown in Figure 11.28. 


AAA 



M M M 


https://www.probabilitycourse.com/chapter11/11_3_4_solved_probs.php 


6/7 








10/8/2018 


Solved Problems 


Figure 11.28 - The transition rate diagram for the above 
queuing system. 


<— previous 
next —> 
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